This example, which is focused on transcriptions, also includes the configuration for a second, unrelated model (voxtral-mini). This could be confusing for users who are looking for a minimal, focused example on how to set up a transcription service.

For better clarity and to make the example easier to copy and adapt, I recommend removing the voxtral_llm_config and updating the app creation on line 80 to only use the whisper_llm_config, like this:

app = build_openai_app({"llm_configs": [whisper_llm_config]})

This will make the example more direct and easier to follow for the specific use case of transcriptions.

I agree, should we just keep voxtral_llm_config which also shows the engine_kwargs?

Reading the entire audio file into memory with await request.file.read() could lead to high memory consumption, especially with large audio files (the OpenAI API limit is 25MB) and concurrent requests. This might risk Out-Of-Memory errors on the replica.

If the underlying create_transcription method in vLLM supports it, consider streaming the file content instead of reading it all at once. This could be done by passing a file-like object or an async generator. If vLLM requires the full byte string, this is an acceptable limitation, but it's an important performance consideration to be aware of.

@eicherseiji a chore we should do is to make this part of setup.py read the llm-requirements.txt directly so we just update one source of truth down the line.

I agree, should we just keep voxtral_llm_config which also shows the engine_kwargs?

I think we should make this method not abstract. So the subclasses can skip their implementation. We can have a NotImplementedError here. I think this points apply to all the endpoints actually, but came to my mind right now :)

Bug: Type Mismatch in Streaming Responses

The LLM*Response type annotations (e.g., LLMChatResponse, LLMCompletionsResponse, LLMTranscriptionResponse) include stream response object types. However, streaming endpoints ultimately yield SSE-formatted strings, not these objects. This creates a mismatch between the declared types and the actual runtime output, which can lead to type checking issues and confusion.

Bug: Request ID Conflict in Transcription Classes

The TranscriptionRequest class explicitly adds a request_id field. This could conflict with vLLMTranscriptionRequest if the base class already defines it, potentially causing duplication or unexpected behavior. This also differs from other request types that inherit request_id from their vLLM base classes.

Bug: Unnecessary Audio Dependencies in Ray[LLM]

The ray[llm] extra now unconditionally pulls in vllm[audio], adding audio-related dependencies. This increases the dependency footprint for all ray[llm] users, even those not needing transcription features, making the installation heavier than necessary.

Bug: Setup.py and Requirements.txt Mismatch

The setup.py adds "meson" and "pybind11" to the llm extras, but these dependencies are not present in the corresponding llm-requirements.txt file (lines 14-15 in the requirements file). While the comment on line 372 states "Keep this in sync with python/requirements/llm/llm-requirements.txt", these two packages are only added to setup.py and not to the requirements file, creating an inconsistency between the two dependency specifications.

Bug: Audio File Reading and Pointer Issue

The code calls await request.file.read() to extract audio data from the request file. However, this reads the entire file into memory, which could be problematic for large audio files. Additionally, after reading the file once, the file pointer is at the end, so if vLLM's create_transcription method tries to read from request.file again, it will get empty data. The audio_data is extracted but the original request.file is still passed to create_transcription, which may cause issues if vLLM expects an unread file object.

Bug: UUID Inconsistency Causes vLLM Compatibility Issues

The TranscriptionRequest adds a request_id with a default UUID, which is inconsistent with other request types that mark this field for upstreaming to vLLM. This approach, questioned in a prior discussion, may cause compatibility issues with vLLM's backend.

Bug: Type Mismatch in Streaming Responses

The LLMChatResponse and LLMCompletionsResponse type annotations include *StreamResponse objects, but the engine's streaming responses actually yield raw strings. This creates a mismatch between the declared types and the actual return values.

Bug: Audio File Serialization Issue in Ray Serve

The transcriptions endpoint uses Annotated[TranscriptionRequest, Form()] to handle form data, but the TranscriptionRequest object contains a file field that is an UploadFile. When this is passed through Ray Serve's remote call mechanism (in _get_response), the UploadFile object may not be serializable for pickling, which is required for Ray remote calls. This could cause serialization errors when the request is sent to the model deployment. The audio data should be extracted and passed as bytes before the remote call, similar to what's done in the vLLM engine at line 473.